# Multimodal Pretraining

Vision model based on SigLIP image encoder, employing global average pooling, part of the PaliGemma project

A vision model based on the SigLIP image encoder, employing global average pooling, and part of the PaliGemma2 project

A vision-language model based on the SigLIP image encoder, utilizing global average pooling, suitable for multimodal tasks.

Vision Transformer model based on SigLIP, containing only the image encoder part, employing a global average pooling strategy

A vision Transformer model based on the CLIP architecture, specifically designed for image feature extraction, released by the LAION organization.

ConvNeXt Base image encoder based on the CLIP framework, trained on the LAION-2B dataset, supports image feature extraction

CLIP image encoder based on ConvNeXt architecture, trained by LAION, suitable for multimodal vision-language tasks

MiniVLA is a vision-language-action model focused on robotics, supporting multimodal tasks from image-text to text.

MiniVLA is a compact yet high-performance vision-language-action model, compatible with Prismatic VLMs training scripts, suitable for robotics and multimodal tasks.

MiniVLA is a lightweight vision-language model compatible with the Prismatic VLMs training framework, supporting multimodal tasks from image-text to text.

CogACT is a novel Vision-Language-Action (VLA) architecture that combines vision-language models with specialized action modules for robotic manipulation tasks.

A vision-language dual-purpose model trained on the LAION-400M dataset, supporting zero-shot image classification tasks

Merlin is a 3D vision-language model for computed tomography, pretrained using both structured electronic health records and unstructured radiology reports.

OpenVLA 7B is an open-source visual-language-action model compatible with Prismatic VLMs training script format, supporting full fine-tuning of 7.5 billion parameters.

LVM is an innovative visual pretraining model that achieves large-scale visual learning by converting visual data into visual sentences and making predictions in an autoregressive manner.

BLIP-2 is a vision-language model that combines an image encoder with a large language model for image-to-text generation and visual question answering tasks.

BLIP-2 is a vision-language model based on OPT-2.7b, which achieves image-to-text generation by freezing the image encoder and large language model while training a query transformer.

MatCha is a vision-language model focused on chart understanding and mathematical reasoning, enhancing processing capabilities through joint modeling of charts and language data

MatCha model fine-tuned on the PlotQA-v1 dataset, specializing in visual question answering tasks for charts, with excellent performance in chart deconstruction and numerical reasoning

MarkupLM is a multimodal pretrained model designed for rich visual document understanding and information extraction tasks, combining text and markup language information.

A specially pretrained English multimodal text encoder based on RoBERTa-base architecture, trained with 1 million image-text pairs

An English MAP visual encoder specially pretrained on COCO and Visual Genome datasets, based on ViT-base architecture

ViLT is a vision-and-language Transformer model pretrained on the GCC+SBU+COCO+VG dataset, focusing on joint understanding tasks of images and text.

A Chinese image-text retrieval model fine-tuned on the COCO-ir dataset based on the Chinese multimodal pretraining model Mengzi-Oscar